Pictogram and iso symbol decoder service

ABSTRACT

This invention provides an instantaneous method for a user or traveler to obtain a meaning of a symbol that is unfamiliar to said user. The symbol is captured in a format that is easily transmitted to a remote database server. Together with the symbol, the GPS coordinates of the location of the symbol must be sent to the server. The server performs an image matching search, and then uses the location information (GPS) to resolve multiple matches and to determine the meaning of the symbol and instantaneously transmits in the language of their choice the meaning to the user requesting the search.

RELATED APPLICATIONS

This a Continuation application of co-pending U.S. patent applicationSer. No. 12/535,242, filed Aug. 4, 2009, the disclosure of which ishereby incorporated by reference herein in its entirety.

FIELD OF DISCLOSURE

This invention relates to a decoder service and more specifically tosymbols used in different countries, including pictograms used on roads.

BACKGROUND

Many countries use pictograms in order to avoid language differences.This is especially true of pictograms used on roads to indicate variousdriving regulations, such as no passing signs, animal crossings, maximumspeeds, etc. Various symbols are present on roads, keyboards, phones,cars, and a myriad of other places. They are meant to overcome languagedifferences; however, in several instances they are difficult tounderstand and may be local in meaning.

With the increase in worldwide tourism and the emerging global market,it is vital that both personal and business visitors have a good graspon road regulations in various countries and communities around theworld. The present invention will provide a vital step in thisdirection.

Unfortunately, there is no standardization of symbols from country tocountry, and the tourist or business visitor is often confused as to themeaning intended to be conveyed. Sometimes a misreading of a road symbolcan lead to accidents because of a difficulty in interpreting the roadsigns. While the present invention can be used as a service to indicatemeanings of symbols in a multitude of foreign sign areas and situations,for the sake of clarity, the present invention will be described as itrelates to symbols on roads relevant to driving on roads in foreigncountries.

Since there are no standard road symbols used worldwide, a service thatwould instantaneously translate the symbols meaning would be extremelybeneficial.

A system for recognizing symbols and converting these symbols into adigital format is described in U.S. Pat. No. 7,508,954 B2, but thesystem described in this patent (Lev) is used primarily forauthentication. The invention utilizes an image of a display showingspecific information which may be open (that is clear) or encoded. Theimaging device captures the image on the display, and a computationalfacility will interpret the information (including prior decoding ofencoded information) to recognize the image. The recognized image willthen be used for purposes such as user authentication, access control,expedited processes, security, or location identification.

The present invention differs from LEV in a major difference in that GPSis used to find the correct locality using the symbol and determines itsmeaning in that locality.

The main thrust of the invention of U.S. Pat. No. 7,508,954 B2 (Lev) isuser identification and access. The purpose appears to be to enablesecure commerce such as buying over the internet, so the user must beidentified from a picture OR a token within the picture must beidentifiable; maybe a coupon, a barcode or a badge. Successfulidentification will allow the transaction to proceed. This is not ageneral look-up service as in the present invention. The patent to Levis about analyzing a digital image or video for specific, predeterminedsymbols or objects. The present invention described herein is thereverse; it deals with general objects and tells what they are. Part oftheir uniqueness is in the algorithms that do the matching (or rathersearching within the image itself). Lev only uses location as part ofthe authentication process. The present invention must use location todetermine the interpretation of the symbol. The Lev patent, the phototaken by the user for authentication represents a proof of the user'slocation which coupled to the user's phone number create reliablelocation-identity authentication. Thus, the location is used in Lev onlyfor user authentication.

The location (GPS coordinates) of the present invention is used todetermine a meaning of a symbol; for example, the same symbol may have adifferent meaning in Italy than in Switzerland. Therefore, in thepresent invention the GPS location of the symbol is critical to thecorrect search conducted and meaning to be determined. This iscompletely different than the location use in Lev.

SUMMARY

Symbols are not always easy to interpret and every region of the worldseems to use different symbols. Symbols related to driving can beespecially confusing and even dangerous if not understood properly.Another area of confusion is pictograms found on machinery or in usermanuals. There have recently been some experiences where the proposedinvention would have come in handy. While driving in France, a foreignvisitor was perplexed by road signs. The GPS that provided directionswould indicate a left turn; however a sign indicated that it wasillegal. Trouble understanding parking signs and other road sings iscommon Here are some examples of confusing symbols:

End of no passing

No passing zone

minimum speed

maximum speed

no parking side 1 on odd days; no parking on side 2 on even days.

oncoming vehicle priority

Another challenge was when a foreign visitor or user needed to replace afuse in a Volkswagen. It was found that the fuse box had an instructioncard, but there were no word descriptions of the symbols and they couldnot find the fuse; one that was suspected might need changing. Otherinstances of frustration are trying to understand symbols on machineinterfaces, cars dashboards and user manuals.

This invention would use a camera equipped cell phone Wi-Fi camera, orscanner to take a picture of the symbol which would be transmitted,wirelessly, along with the GPS coordinates, to a service that would usethe information to look up the symbol in a set of databases andimmediately provide the meaning back to the user in a verbal or textformat. Wi-Fi cameras have better resolution than cell phone cameras andcan automatically transfer an image over a network. There are severalappropriate cameras on the market today. Cell phone camera resolutionscontinue to improve. Some older versions may not be totally adequate forthis service but newer versions are usable and future ones will be evenbetter. Originally the phone cameras where 1 megapixel. Many phonestoday are over 3 megapixels with some as high as 8 megapixels. Likewise,cell phone networks are getting much faster and will continue to do soand will function in the present invention.

This invention relies on the following capabilities:

The decoder service would need a large database of pictures from aroundthe world with images of pictograms, pictographs and iso symbols. Thedatabase would need to be capable of being updated on a regular basis asnew images are identified. The interpretation of the symbol would bepart of the metadata stored with the image. The database may alsoinclude metadata that would be useful in interpreting the image. Forexample, if a user was in France or Germany and I snapped an image of aroad sign, the GPS coordinates of my location can be used todifferentiate a symbol that has different meanings in differentcountries. There are many online collections of iso-symbols and roadsign images that can be used to create the database(s).

Imaging software that could find the area of the photograph or scan thatwas of interest and crop everything else out. Many digital cameras comewith this capability to identify an area of interest, such as a face,and Xerox has developed software called “FaceFinder” that is used inXerox imaging center to process photo IDs for credit card applications.It can find the face in the image and then enlarge or shrink the imageto the size needed for the credit card. Similarly, in this invention wewould need to identify and extract the pictogram image from thephotograph or scan. In the case of multiple images, it could recognizeand separate multiple pictograms or it can provide a feature to allowthe user to select the specific image it needs information on. If thesystem needed to return the definition of several images then it wouldhave to show the image to the user along with the interpretation toavoid any confusion. Additional image clean up and enhancement would benecessary to center the image, straighten it up, remove noise (salt andpepper clean up), etc.

The service would use an imaging recognition and categorizing softwareto search the database. There are commercial software packages that canbe trained to do this and Xerox has its own software that can be used,i.e., GVC (Generic visual categorizer) and/or DICE (Document imageclassification engine).

Image acquisition can come from many sources as long as it produces adigital file. Cell phone cameras or regular digital cameras can be usedprovided that caution is exercised while driving. Paper documents can bescanned in or electronic documents can be submitted to the Decoderservice. This application is ideally suited as a service rather than aclient application because of the requirement of a large, up to dateimage library and the sophistication of the software necessary to cleanup the images and to perform the image search. The system should alsoprovide a way to research images that could not be found in the databaseso that it can be updated.

A novelty of this invention is in construction of the database ofsymbols and in the application of existing technologies in imageprocessing and visual search to a practical problem and in the creationof a service that real people can access in a simple manner usingcommonly available equipment such as network connected cell phones andlaptop computers.

All of the image processing software is available and in use as is thevisual categorization and search software. It would need to be trainedwith an appropriate image set. The database has not been created butthere are many different on line collections of pictograms andiso-symbols. It would require work to generate but the information isavailable.

There is some work being conducted today that could be used inestablishing the database of this invention for example the UCSC Centerfor Research in Language is engaged in a large international study toprovide an index of symbols in several different languages—American,English, German, Mexican, Spanish, Italian, Bulgarian, and Hungarian. Apublication IS07001 (Public information symbols) is a standard publishedby the International Organization for Standardization that defines a setof pictograms and symbols for public information. A latest version ofthis publication is IS07001:2007, November 2007. This effort, whilehelpful, is concerned with standardizing various symbols; this would‘not be necessary with the development of the database of the presentinvention.

The present invention leverages some existing services businessinfrastructure and the imaging technology that has been developed. Thereis a lot of work being done in visual search technologies but most aretaking a specific word and searching for images based on that work.Shahi (http://blachan.com/shahil) is a visual dictionary that combinesWiktionary content with Flickr images, and more) and others are used toautomatically categorize photos based on what is in the photo (Viewzi;SearchMe; uvLayer and oSkope). There is no known specific tool thatanalyzes and interprets pictograms or iso-symbols. The advantages ofhaving this invention should be clear. Even if one had a hard copyreference guide, there is no simple way to index the guide to find anarbitrary image. This type of search, of an image against and imagedatabase is best handled by computers. Being able to have this type ofinformation when traveling has implications on safety, avoiding costlymistakes and general frustration. There are no technical obstacles tothe present invention to solving this problem, it is a matter ofcollecting the right master data set, training the tools to the data setand setting up the interface using internet technology.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart illustrating the steps in an embodiment of thisinvention.

DETAILED DESCRIPTION

In FIG. 1 a pictogram decoding service is provided using your cell phonecamera, wi-fi camera, scanner or a computer file as input. Take apicture of the symbol and select the decode service which may residesomewhere on the internet. The service utilizes the image and your GPScoordinates to determine the meaning of the symbol which it can provideto you as a text message or a verbal response. The GPS coordinates is acritical part of the present process and invention, because without it,the meaning in that locality cannot be determined.

In FIG. 1 an outline of the steps in the present process areillustrated. A symbol 1 that is confusing is photographed or imaged by acell phone 2, digital camera 10, scanner 3 or computer 4, the imagetaken is improved to the extent possible by the steps of clean up image5. The GPS coordinates 6 of the location of the symbol 1 is determined,and the cleaned up image 5 and the GPS 6 are forwarded to a databaseserver 7. The database server 7 compares and searches at 9 the image 5conveyed within the GPS 6 location and as soon as possible withinseconds returns results to the user as shown at 8. If the response tothe results of the search is not within about 10-15 seconds, theusefulness of this process is reduced. This response is defined as “ashort time” in the claims. Usually the response could be in the form ofa call to the user cell phone, or a text message to the user on themeaning of the symbol. Without the searcher's knowledge of the GPScoordinates, the search of symbol meaning within that specific areawould be meaningless.

Therefore, in summary, the two critical features to be in the presentinvention are:

1. The proper database that can isolate and identify your symbol forreview;

2. The GPS coordinate where the photo of the symbol was taken to get amatch on the symbol. An immediate verbal or oral response via phone ispreferred. Without either of these criteria the process of the presetinvention will not function.

This invention provides a method of identifying symbols includingtraffic pictograms which comprises a user capturing the image or symbolto be identified by an imaging device selected from the group consistingof cell phone camera, digital camera, wi-fi camera, scanner or computerfile then transmitting this information to a database server includingthe image from the imaging device and user's GPS location where thesymbol exists to obtain the local meaning of the symbol.

The server will process the information that was transmitted includingthe image or symbol and the user's location re GPS coordinates for theserver to determine a meaning of the symbol within the location wherethe user captured the image.

The server will compare the symbol imaged that was transmitted withsymbols in the GPS indicated location in order for the server todetermine the meaning of the symbol and subsequently transmit from thedatabase server to the user the meaning of the symbol in the locationidentified by the GPS. The image may be cleaned up to be clearer priorto transmission by the user to the database server. The database serverutilizes image recognitions and categorizing software and the GPS tosearch in the database.

In some cases, images that are not present in the database server willbe added to the database to update the database for enhanced futureusage. The database comprises a collection of pictograms and iso-symbolseach existing with precise meanings within various locationsidentifiable by GPS coordinates. The image acquisition is produced indigital format for searching in the database. This should have aWikipedia-like interface so that users can edit the definitions toimprove their accuracy or to identify symbols that the system is notfamiliar with. The system should be able to accept verbal or audio inputto add to the database. Also, the database must be multilingual so thatthe service is available worldwide.

The database has symbols categorized by the GPS locations where theyexist and the GPS locations are matched with a requested search todetermine symbol meaning. The database server in one embodiment isconfigured to respond to the user in a short time via oral communicationto a user cell phone. This is meant to be a general purpose symboldecoder, not just for road signs. Other symbols could be found onproduct packaging, or within documentation. Electrical symbols,construction diagram symbols, or even hieroglyphics or heraldic symbols.

It will be appreciated that variations of the above-disclosed and otherfeatures and functions, or alternatives thereof, may be desirablycombined into many other different systems or applications. Variouspresently unforeseen or unanticipated alternatives, modifications,variations, or improvements therein may be subsequently made by thoseskilled in the art which are also intended to be encompassed by thefollowing claims.

What is claimed is:
 1. A method of identifying symbols including trafficpictograms for use with a GPS device, the method comprising: receivingimage information from an imaging device, the image informationincluding an image and a GPS location corresponding to a location ofwhere the image was captured; and processing said information at adatabase server located geographically remotely from the location ofimage capture, the information received including said image informationand said GPS location to determine a meaning of said image within saidlocation where said user captured said image.
 2. The method of claim 1,the processing further comprising: comparing said captured image withstored images or symbols to determine that the captured image matches astored image or symbol, the stored images or symbols being associatedwith the GPS location.
 3. The method of claim 1, comprising:transmitting from said database server to said user device said meaningof said symbol at the GPS location, the meaning being associated withthe stored image or symbol determined to match the captured symbol. 4.The method of claim 2, wherein when a match is not determined, storingthe captured image in the database server.
 5. The method of claim 1,wherein said database comprises a collection of pictograms andiso-symbols each existing with precise meanings within various locationsidentifiable by GPS coordinates.
 6. The method of claim 1, wherein thecaptured image is a digital image.
 7. The method of claim 1 wherein theimages or symbols of said database are categorized by the GPS locationswhere they exist.
 8. The method of claim 1 wherein said database serveris configured to respond to said user at a user's mobile telephone orpersonal electronic device.
 9. The method of claim 2, said comparingcomprising comparing an image appearance with a stored image appearanceand/or comparing a captured image GPS location with a stored image orsymbol GPS location.
 10. A system for determining a meaning of symbols,comprising: an imaging device for capturing for capturing an image of asymbol using an imaging device selected from a group consistingessentially of a cell phone camera, digital camera, wi-fi camera, andscanner, the imaging device comprising a GPS system; an image database,the image database being configured to store image data and associatedGPS location data, the imaging system being configured to transmit imagedata to the image database, the image data corresponding to the image ofthe symbol, and captured image location data, the location datacomprising a geographical location of the imaging device; and aprocessor configured for determining a meaning of the captured imagebased on the image data and the location data.
 11. The system of 10, thedetermining further comprising: comparing the captured image data withstored image or symbol data for determining whether the captured imagematches an image or symbol stored in the database, or a location of animage or symbol stored in the database.
 12. The system of claim 10, thedetermining further comprising: updating the stored image or symbol datato include the captured image when the stored image or symbol data doesnot include the captured image before the updating.
 13. The system ofclaim 10, the image database being configured to store symbol data, thesymbol data including at least one of a pictogram and an iso-symbol, andat least one meaning corresponding to the pictogram or the iso-symbol.14. The system of claim 10, the meaning being associated with ageographical location.
 15. The system of claim 10, the determiningcomprising searching the database for symbol data based on the locationdata.
 16. The system of claim 10, comprising: a user cell phone, theuser cell phone being configured to receive a determined meaning of acaptured symbol at the user's cell phone.
 17. The system of claim 10,the processor being configured to cause the determined meaning of thecaptured symbol to be transmitted to a user's cell phone.
 18. The systemof claim 10, comprising: a user's mobile device, the mobile device beingconfigured to receive a determined meaning of a captured symbol at theuser's mobile device.
 19. The system of claim 10, wherein the imagedatabase is geographically remotely located from the imaging deviceduring the determining.
 20. The system of claim system of claim 18,wherein the user's mobile device is configured for acquiring a globalposition of the mobile device, and associating the determined globalposition with an image captured by the imaging device.